Network anomaly detection and network performance status determination

ABSTRACT

A system may collect, from a wireless network, first data pertaining to nodes in the wireless network. Each datum of the first data belongs to one of two or more categories/For each of the nodes, for each of the categories, and for each datum belonging to the category, the system may determine if the datum is outside of a first range of values, and if the datum is inside the first range, the system may calculate a first base network performance health (NPH) score that is a function of the nodes, the categories, the data, and time. The system may also apply first deep learning to a first neural network among a plurality of neural networks to update first coefficients for correlating the first base NPH score to a mean opinion score, for each of the categories.

RELATED APPLICATION

The present application is a continuation of, and claims priority to,U.S. application Ser. No. 15/880,855 filed Jan. 26, 2018, the contentsof which are incorporated by reference herein in their entirety.

BACKGROUND INFORMATION

Past years have witnessed exponential growth of network users in theworld. For example, global Long-Term Evolution (LTE) subscribers reacheda total of 2.1 billion by the first quarter of 2017. Such staggeringgrowth in the number of subscribers drives network traffic to risequickly. Global mobile data traffic grew 63% in 2016, reaching 7.2exabytes per month at the end of 2016, up from 4.4 exabytes per month atthe end of 2015. Mobile data traffic in the world grew 180-fold over thepast 5 years.

The snowballing effect of the mobile traffic growth has overwhelmeddifferent data and voice networks. Data tsunami and signaling stormshave been swamping these networks, and challenging existing cellulartechnologies. The traffic is still growing, along with users' demands onnetwork performance.

For network service providers, the continued mobile traffic growth meansthe continued expansion of network capacity and the optimization ofnetwork performance so as to maintain the quality of service (e.g.,throughput and latency) at an optimal level that is perceivable byusers. Steps need to be taken in order to prepare the networks andbusiness operations for the challenges of upcoming technology, such as5G networks and Internet-of-things.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates an exemplary output of an intelligent anomalydetector according to one implementation;

FIG. 1B illustrates a screenshot of an exemplary output associated withan exemplary network performance health monitor according to oneimplementation;

FIG. 1C illustrates an exemplary network reconfiguration system includedin a base station according to one implementation;

FIG. 2 illustrates an exemplary network environment in which theconcepts described herein may be implemented;

FIG. 3 illustrates exemplary components of network devices included inthe network environment of FIG. 2;

FIG. 4 shows exemplary components of a network performance analysissystem of FIG. 2;

FIG. 5 shows exemplary components of the exemplary intelligent anomalydetector of FIG. 4;

FIG. 6 shows an exemplary box plot of input data to the intelligentanomaly detector of FIG. 5;

FIG. 7A illustrates exemplary interquartile ranges (IQRS) associatedwith input data;

FIG. 7B illustrates an exemplary output of the intelligent anomalydetector of FIG. 4 and FIG. 5;

FIG. 8A shows an exemplary plot of input data as a function of time;

FIG. 8B shows an exemplary trend component associated with the plot ofFIG. 8A;

FIG. 8C shows an exemplary seasonal component associated with the plotof FIG. 8A;

FIG. 8D shows an exemplary plot of data whose trend and seasonalcomponents have been removed;

FIG. 9 illustrates an exemplary plot of input data for the exemplaryleast-squares error model selector of FIG. 5;

FIG. 10 is a flow diagram of an exemplary process that is associatedwith the intelligent anomaly detector of FIG. 4 and FIG. 5;

FIG. 11 illustrates an exemplary process that is associated with theexemplary network health monitor of FIG. 4;

FIG. 12 illustrates logic associated with the process of FIG. 11;

FIGS. 13A and 13B illustrate areas under a curve (AUC) for a keyperformance indicator (KPI);

FIG. 14 depicts training exemplary convolutional neural networks (CNNs)in the process associated with FIG. 11 and FIG. 12;

FIG. 15 is a flow diagram of a process for determining an exemplary basenetwork performance health (NPH) score for an exemplary grid element;

FIG. 16A is a flow diagram of a process for determining exemplarybi-weekly base NPH scores for network nodes in the network environmentof FIG. 2;

FIG. 16B is a flow diagram of a process for determining exemplarybi-weekly base NPH scores for exemplary grid elements for the networkenvironment of FIG. 2;

FIGS. 17A and 17B illustrate processing daily base NPH scores to obtainbi-weekly NPH scores for network nodes in accordance with the processesof FIG. 16A according to one implementation;

FIG. 17C illustrates processing daily base NPH scores to obtainbi-weekly NPH scores for grid elements in accordance with the processesof FIG. 16B according to one implementation;

FIG. 18 is a flow diagram of a process for determining exemplaryperformance adjusters, final daily NPH scores, and final bi-weekly NPHscores for network nodes and grid elements in the network environment ofFIG. 2; and

FIG. 19 illustrates processing that is associated with determiningperformance adjusters of FIG. 18;

FIG. 20 illustrates exemplary final NPH scores and final bi-weekly NPHscores for different types of nodes;

FIG. 21 illustrates an exemplary decision tree that is associated withNPH scores; and

FIG. 22 is a block diagram of exemplary functional components of thenetwork reconfiguration system of FIG. 1C and FIG. 4.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The following detailed description refers to the accompanying drawings.The same reference numbers in different drawings may identify the sameor similar elements.

As described herein, a network includes a network performanceoptimization system. The system, in turn, comprises an intelligentanomaly detector, a network performance health monitor, and a networkreconfiguration system. The intelligent anomaly detector detects and/orpredicts potential network anomalies. Anomaly detection (i.e., outlierdetection) includes the process of identifying events, observations,objects, and/or characteristics which do not conform to expected networkpatterns. An anomaly detector for a wireless network typically operatesin a reactive mode. Accordingly, an anomaly in a network must firstoccur before the anomaly detector reports the anomaly. No matter howtimely the report, the anomaly is reported after the fact, i.e., afterthe anomaly occurs since the detector operates in the reactive mode. Incontrast, as described herein, an intelligent anomaly detector learnsthe latest traffic patterns and their characteristics to identify safetybands of operation and proactively identifies and reacts to anomalies inadvance.

In addition to the intelligent anomaly detector, the network performanceoptimization system includes a network performance health monitor thatfocuses on user Quality of Experience (QoE). Traditional networkoptimizations are driven by performances of individual network elements.Network elements such as cells, eNodeBs, mobility management entities(MMEs), serving gateways (SGWs), packet data network gateways (PGWs),etc., are typically targeted to be optimized based on their operationalparameters. Such optimizations, however, do not address users' QoE.

As described herein, a network performance health monitor extends thegoal of achieving optimum network performance to maximizing user QoE.The network performance health monitor increases user QoE by tyingtogether user perception, objective experience, and expectations toapplications and network performance measured by network key performanceindicators (KPIs).

The network performance optimization system further includes a networkreconfiguration system. The network reconfiguration system receives theoutput from the intelligent anomaly detector and network performancehealth monitor. Based on the output, the network reconfiguration systemmodifies network settings and operating parameters, including but notlimited to those related to antenna (e.g., tilt angles, beam width,transmission power, etc.), routing paths, load balancing, bandwidths,etc.

FIG. 1A illustrates an exemplary output 100 of an intelligent anomalydetector according to one implementation. As shown, output 100 includesa plot 102 of call drop rates as a function of time. When theintelligent anomaly detector receives input data (i.e., call droprates), the intelligent anomaly detector filters outliers 106 from thedata, and uses the filtered data to generate an upper bound 104. Anypoint, on plot 102, that exceeds upper bound 104 may be subsequentlydetected as an anomaly. In addition, any predicted point (not shown)that exceeds upper bound 104 may be detected as a potential anomaly.

FIG. 1B illustrates a screenshot 110 of an output associated with thenetwork performance health monitor. As shown, screen shot 110 includes agrid that is overlaid on a map, along with images of cells (e.g., cell112). For each cell 112, each grid element (i.e., a shaded box), or eachof other types of network elements (not shown), the network performancehealth monitor may determine network performance health scores thatapproximate true user QoE. Each health score may be determined for aspecific technology (e.g., Long-term evolution (LTE), Voice-over-LTE(VoLTE)) and/or for a specific time period, such as a day, a week,2-weeks, a month, etc.

FIG. 1C illustrates an exemplary network reconfiguration system 414 inan exemplary network. As shown, network reconfiguration system 114 mayinclude a reconfiguration manager 2202 and a network node 116. Althoughnetwork node 116 is illustrated as a base station in FIG. 1C, node 116may include another type of node, such as a router, a load balancer, aswitch, a gateway, a server (e.g., Domain Name Server), or another typeof network element.

Reconfiguration manager 2202 may receive data indicating detected oranticipated anomalies and network performance health scores from theintelligent anomaly detector and network performance health monitor,respectively. Furthermore, based on the received information,reconfiguration manager 2202 may send data (e.g., regarding anomaliesand health scores) and/or different commands to node 116, to changedevice settings at node 116 or its operating parameters to new values.

For example, assume that an antenna installed at base station 116 istilted at a particular angle to accommodate an efficient handoff fromanother base station. Also assume that data pertaining to ananticipated/detected network anomaly on traffic and/or networkperformance health scores indicates that tilting the antenna in aslightly different direction may eliminate the potential anomaly andimprove the health scores. In this example, network reconfigurationmanager 2202 may send the data and/or a command to base station 116. Inresponse, base station 116 may modify the direction of its antenna. Fora network node different from a base station, network reconfigurationmanager 2202 may send different data and/or commands from those for basestations, and such a node may respond in ways consistent with itsfunction in the network.

FIG. 2 illustrates an exemplary network environment 200 in which anetwork performance optimization (NPO) system described herein may beimplemented. As shown, environment 200 may include a user device 202, aprovider network 204, and a packet data network (PDN) 206. Althoughenvironment 200 may include other devices and components, forsimplicity, they are not illustrated. For example, environment 200 mayinclude millions of user devices, routers, switches, computers (e.g.,servers, personal computers, etc.).

User device 202 may include an electronic device having communicationcapabilities. For example, user device 202 may include a smart phone, awearable computer (e.g., a wrist watch, eye glasses, etc.), a tablet, aset-top box (STB), any type of internet protocol (IP) communicationsdevice, a voice over internet protocol (VoIP) device, a laptop computer,a palmtop computer, a gaming device, a media player device, a digitalcamera that includes communication capabilities (e.g., wirelesscommunication mechanisms), or Internet-of-Things (IoTs).

Provider network 204 may include one or more wireless and/or wirelinenetworks of any type, such as, for example, a local area network (LAN),wide area network (WAN), wireless local area network (WLAN), wirelesssatellite network, and/or one or more wireless public land mobilenetworks (PLMNs). The PLMN(s) may include a Code Division MultipleAccess (CDMA) 2000 PLMN, Global System for Mobile Communications (GSM)PLMN, Long Term Evolution (LTE) PLMN and/or other types of PLMNs notspecifically described herein.

Depending on the implementation, provider network 204 may include an LTEnetwork that includes an evolved UMTS Terrestrial Network (eUTRAN) 208.eUTRAN 208 may include one or more of eNodeB 210. eNodeB 210 may includeone or more devices and components that allow user device 202 towirelessly connect to provider network 204. Although referred to hereinas eNodeB 210, in other network environments, features and functionalityassociated with eNodeB 210 may be implemented or performed by similarbase station devices. Each of such devices may transmit to or receivesignals from a specified area, herein referred to as a “cell.”

As shown, provider network 204 may also include a network performanceoptimization (NPO) system 212, which in turn may include an intelligentanomaly detector, a network performance health (NPH) monitor, and anetwork reconfiguration system (not shown). As indicated above, theintelligent anomaly detector may detect or predict network anomalies.The NPH monitor may determine NPH scores for various network elements, aspecific geographical area (e.g., an area represented by a grid elementin FIG. 1B), a specific category of key performance indicator, for aspecific time interval, and/or specific technology (e.g., LTEtechnology, VoLTE technology, etc.). The network reconfiguration systemmay modify operating parameters of network components based on thedetected/anticipated anomalies and NPH scores.

Packet data network (PDN) 206 may include a network that supportsInternet Protocol (IP)-based communications. PDN 206 may include, forexample, an IP Multimedia Subsystem (IMS) network, which may providevoice and multimedia services to user device 202 based on the SessionInitiation Protocol (SIP).

Although FIG. 2 shows NPO system 212 as a single component, depending onthe implementation, NPO system 212 may be distributed over differentnetwork elements and different networks. For example, NPO system 212 mayinclude network devices that collect user device information. In otherembodiments, NPO system 212 may include user devices 202. In still otherimplementations, NPO system 212 may include network management stations(NMS). NPO system 212 may be designed in accordance with a centralizedarchitecture or a decentralized architecture. NPO system 212 isdescribed further below with reference to FIGS. 4-22.

FIG. 3 is a block diagram of exemplary components of a network device300. Network device 300 may correspond to, or be included in, thedevices and/or components of the networks depicted in FIG. 2 (e.g., userdevice 202, a router, a switch, a server, etc.). In some embodiments,NPO system 212 may be implemented on one or more of network device 300.As shown, network device 300 may include a processor 302, memory/storage304, input component 306, output component 308, network interface 310,and communication path 312. In different implementations, network device300 may include additional, fewer, different, or a different arrangementof components than the ones illustrated in FIG. 3. For example, networkdevice 300 may include line cards, modems, etc.

Processor 302 may include a processor, a microprocessor, an ApplicationSpecific Integrated Circuit (ASIC), a Field Programmable Gate Array(FPGA), programmable logic device, chipset, application specificinstruction-set processor (ASIP), system-on-chip (SoC), centralprocessing unit (CPU) (e.g., one or multiple cores), microcontrollers,and/or other processing logic (e.g., embedded devices) capable ofcontrolling device 300 and/or executing programs/instructions.

Memory/storage 304 may include static memory, such as read only memory(ROM), and/or dynamic memory, such as random access memory (RAM), oronboard cache, for storing data and machine-readable instructions (e.g.,programs, scripts, etc.).

Memory/storage 304 may also include a floppy disk, CD ROM, CD read/write(R/W) disk, optical disk, magnetic disk, solid state disk, holographicversatile disk (HVD), digital versatile disk (DVD), and/or flash memory,as well as other types of storage device (e.g., Micro-Electromechanicalsystem (MEMS)-based storage medium) for storing data and/ormachine-readable instructions (e.g., a program, script, etc.).Memory/storage 304 may be external to and/or removable from networkdevice 300. Memory/storage 304 may include, for example, a UniversalSerial Bus (USB) memory stick, a dongle, a hard disk, off-line storage,a Blu-Ray® disk (BD), etc. Memory/storage 304 may also include devicesthat can function both as a RAM-like component or persistent storage,such as Intel® Optane memories.

Depending on the context, the term “memory,” “storage,” “storagedevice,” “storage unit,” and/or “medium” may be used interchangeably.For example, a “computer-readable storage device” or “computer-readablemedium” may refer to both a memory and/or storage device.

Input component 306 and output component 308 may receive input from auser and provide output to a user. Input/output components 306 and 308may include, for example, a display screen, a keyboard, a mouse, aspeaker, a microphone, a camera, a DVD reader, USB lines, and/or othertypes of components.

Network interface 310 may include a transceiver (e.g., a transmitter anda receiver) for network device 300 to communicate with other devicesand/or systems. For example, via network interface 310, network device300 may communicate over a network, such as the Internet, an intranet, aterrestrial wireless network (e.g., a WLAN, WiFi, WiMax, etc.), asatellite-based network, optical network, etc. Network interface 310 mayinclude a modem, an Ethernet interface to a LAN, and/or aninterface/connection for connecting device 300 to other devices (e.g., aBluetooth interface).

Communication path 312 may provide an interface (e.g., a bus) throughwhich components of device 200 can communicate with one another.

In some implementations, network device 300 may perform the operationsdescribed herein in response to processor 302 executing softwareinstructions stored in a non-transient computer-readable medium, such asmemory/storage 304. The software instructions may be read intomemory/storage 304 from another computer-readable medium or from anotherdevice via network interface 310. The software instructions stored inmemory/storage 304, when executed by processor 302, may cause processor302 to perform processes that are described herein. For example, whennetwork device 300 is implemented as user device 202, user device 202may collect network data and provide the data to provider network 204.In another example, when network device 300 is implemented as NPO systemdevices, network device 300 may detect network anomalies or calculateNPH scores in accordance with program instructions. In yet anotherexample, implemented as a network node (e.g., a load balancer) capableof responding to network reconfiguration manager 2202, network device300 may re-route packets that require specific quality-of-service (QoS).

FIG. 4 illustrates exemplary components of NPO system 212. As shown, NPOsystem 212 may include a data collection system 402, a data platform404, a network analytics engine 406, and a data visualization andautomation system 408. Depending on the implementation, NPO system 212may include additional, fewer, different, or differently arrangedcomponents than those illustrated in FIG. 4.

Data collection system 402 may include components for obtaining networkdata, network statistics (e.g., traffic, delays, jitter, bandwidth, calldrop rates, handoffs, bearer identifier, service availability, etc.),Key Performance Indicators (KPIs), user mean opinion scores (MOS), etc.The information may be obtained from user devices, Internet-of-Things(IoT) devices, eNodeBs, server devices, routers, switches, applications,servers, service components (e.g., MME, PGW, etc.), etc. Some of thedata may include KPIs pertaining to device statistics, network trafficdata (e.g., volume, location at which the measurements are taken), time,jitter, delay/latency, power (e.g., signal-to-interference and noiseratio), antenna characteristics, call drop rates, applicationcharacteristics (e.g., TCP connection request success rate), etc.

Data collection system 402 may collect data at regular time intervals(e.g., according to a schedule), based on demand, and/or based on eventtriggers. Data collection system 402 may provide the collected data todata platform 404 through one or more interfaces, such as a MySQLinterface, a JSON interface, a JDBC interface, etc.

Data platform 404 may receive data from data collection system 402,process some of the data to generate statistics, store the raw orprocessed data, and make the data available for other components, suchas network analytics engine 406 or data visualization and automationsystem 408. In storing and/or processing data, data platform 404 mayleverage one or more big data platforms.

Network analytics engine 406 may include many components with logic fordetermining and computing network parameters, including those for:identifying network anomalies based on learning; generating NPH scores;modifying neural network (e.g., calculating neural network weights);etc.

Data visualization and automation system 408 may perform outputfunctions for data platform 404 and network analytics engine 406. Forexample, data visualization and automation system 408 may provide outputsimilar to those illustrated in FIG. 1A or 1B. In addition, datavisualization and automation system 408 may allow users to inputinformation into NPO system 212, control different components anddevices of NPO system 212 and provider network 204, administeraccounts/other users, etc. In some implementations, data visualizationand automation system 408 may provide different forms of network alertsto operators (e.g., send a message to a network operator), takeappropriate actions with respect to network (e.g., change devicesettings), reallocate resources (e.g., bandwidth allocation, recalibrateload balancing systems, etc.), reconfigure network components, routepackets through particular paths, etc.

As further shown in FIG. 4, NPO system 212 may include an intelligentanomaly detector 410. network performance health (NPH) monitor 412, andnetwork reconfiguration system 414. Each of intelligent anomaly detector410, NPH monitor 412, and network reconfiguration system 414 may use orincorporate data visualization and automation system 408, networkanalytics engine 406, and data platform 404. Intelligent anomalydetector 410, NPH monitor 412, network reconfiguration system 414 aredescribed further below with references to FIGS. 5-22.

FIG. 5 shows exemplary functional components of intelligent anomalydetector 410. As shown, intelligent anomaly detector 410 may include apre-processor 504, an ensemble learning anomaly detector 506, and anInterquartile (IQR) detector 508. Pre-processor 504, in turn, mayinclude an extreme value detector 510 and a white-noise detector 512.Ensemble learning anomaly detector 506, in turn, may include a basicdecomposition module 514, a seasonal autoregressive integrated movingaverage (ARIMA) module 516, a decomposition based ARIMA module 518, anda least squares error model selector 520. In operation, intelligentanomaly detector 410 receives user input from user interface 502-1 andprovides output to user interface 502-2.

Pre-processor 504 receives data (e.g., KPIs) from data platform 404 andprepares the data for further processing. The preparation may includefiltering extreme-valued data points, by applying extreme value detector510, and then determining whether the filtered data is essentially whitenoise, by applying white-noise testing module 512. Depending on theimplementation, pre-processor 504 may include additional functionalcomponents required to further process data.

Extreme value detector 510 may remove data points whose values areextreme (e.g., higher or lower-valued than the rest of the data set by agiven amount). According to one implementation, extreme value detector510 may divide the data into interquartile ranges, and classify datapoints outside of specific bounds as “extreme.” FIG. 6 illustrates theconcept with an exemplary box plot of data. As shown, data (withoutextreme values) spans a range 602 bounded by the maximum value 606 andthe minimum value 614. 50% of data fall in the interquartile range 604,with 25% of data above the 75 percentile 608 (Q₃) and 25% of data belowthe 25 percentile 612 (Q₁). Extreme values 616 fall outside range 602and satisfy threshold criteria. For example, in one implementation,extreme values must be larger than about 1.5 IQR+Q₃ or less than aboutQ₁−1.5 IQR. After extreme value detector 510 detects data points withextreme values and filters them from the data set, extreme valuedetector 510 provides the filtered data to white-noise detector 512.

White-noise detector 512 may determine whether the filtered data iswhite noise. White noise is completely random, and there is norelationship between any two points in the data set. In oneimplementation, white-noise detector 512 may use the Ljung-Box test todetermine whether the filtered data is white noise. In otherimplementations, white-noise detector 512 may use a different test todetermine whether the data is white noise (e.g., Bartlett test).

IQR anomaly detector 508 may detect anomalies in a data set ifwhite-noise testing module 512 determines that the filtered data iswhite noise. IQR anomaly detector 508 identifies data points that lieoutside of a specific range of values as anomalies. FIG. 7A illustratesthe concept. As shown, data that lie within the range [Q₁, Q₃] (thevalues of Q₁ and Q₃ are defined so that the range [Q₁, Q₃] includes 50%of data) is deemed normal, and data that lie outside of the range[Q₁−1.5 IQR, Q₃+1.5 IQR] are anomalies. As also shown, Q₁ isapproximately equal to −0.674 σ and Q₃ is approximately equal to 0.674σ, where σ denotes the standard deviation of the illustrated function.

FIG. 7B illustrates an exemplary plot 702 of data to which IQR anomalydetector 508 has been applied. As shown, unfiltered data is plotted withan upper bound 704, also obtained by applying IQR anomaly detector 508.Data points whose values are above the upper bound by a specified marginare deemed anomalies. A lower bound is not illustrated.

Returning to FIG. 5, ensemble learning anomaly detector 506 may detectanomalies in the filtered data when white noise testing module 512determines that the filtered data is not white noise. In such a case,ensemble learning anomaly detector 506 may use a basic decompositionmodule 514, a seasonal ARIMA module 516, a decomposition based ARIMAmodule 518, or a least-squares error model selector 520 to detect orpredict anomalies. Which of modules 514-520 is selected depends on userinput provided through user interface 502-1.

FIGS. 8A-8D illustrate operations of basic decomposition module 514. Asits name implies, basic decomposition module 514 decomposes input datainto, for example, three components: a trend component, a seasonalcomponent, and a noise component. That is, the input data isapproximated as:I(t)=T(t)+S(t)+N(t)  (1),where I(t) is the input data, T(t) is the trend component, S(t) is theseasonal component, and N(t) is the noise component.

FIG. 8A is a plot 802 of I(T). In FIG. 8A, I(t) increases as a functionof time in an undulating manner. FIG. 8B illustrates a plot of T(t),which is a trend component 804 that has been isolated from input data802. T(t) increases monotonically over the shown period of time.Although shown as a line, in different implementations, T(t) may be anonlinear function. FIG. 8C shows a plot of S(t), which is a seasonalcomponent 806 that has been isolated from I(t). S(t) may be cyclic,having a period after which S(t) repeats. FIG. 8D shows a plot of N(t),which is a noise component 808 that has been isolated from I(t). N(t)may be obtained by removing or subtracting T(t) and S(t) from I(t).

Returning to FIG. 5, seasonal ARIMA module 516 models input data asbeing composed of a non-seasonal component and a seasonal component.This can be expressed as:y _(t) =y _(s) +y _(n)  (2),where y_(s) is the seasonal component and y_(n) is the non-seasonalcomponent. y_(s) and y_(n) are defined by different expressions:y _(n)(t): y′ _(t) =y _(t) −y _(t-1)y _(s)(t): y′ _(t) =y _(t) −y _(t-m)where m is the number of seasons. After the seasonal component isremoved from the data set, the remaining data can then be modeled as anautoregressive moving average (ARMA):

$\begin{matrix}{{X(t)} = {c + ɛ_{t} + {\sum\limits_{i = 1}^{p}{\phi_{i}X_{t - i}}} + {\sum\limits_{i = 1}^{q}{\theta_{i}ɛ_{t - i}}}}} & (3)\end{matrix}$In expression (3), c is a constant, ε_(t), ε_(t-1), . . . are whitenoise, ϕ_(i) and θ_(i) are parameters of the ARMA model. The sum overX_(t-i) is the autoregressive term, and the sum over θ_(i) is the movingaverage term. In one implementation, data points that significantlydeviate from the predicted values can be identified as anomalies.

Decomposition based ARIMA module 518 combines the features of basicdecomposition module 514 and seasonal ARIMA module 516. Seasonal ARIMAmodule 516 is useful for prediction, but its results may not be easy tointerpret. Basic decomposition module 514 generates results that areeasy to interpret, but may not use some information in noise component808 that can be used for making more accurate predictions. The functionof decomposition based ARIMA module 518 can be summarized as:X(t)=T(t)+S(t)+A(t)where T(t) is a trend component, S(t) is a seasonal component, and A(t)is an autoregressive component, Decomposition based ARIMA module 518applies basic decomposition module 514 to remove trend and seasonalcomponents, and then applies an autoregressive model for prediction offuture values.

Least-squares error model selector 520 applies the results or output ofbasic decomposition module 514, seasonal ARIMA module 516, anddecomposition based ARIMA module 518. Subsequently, selector 520determines a set of coefficients for forming a linear combination, ofthe outputs of modules 514-518, that leads to least-squares errorsbetween predicted values and the input. FIG. 9 illustrates an exemplaryplot of data input to least-squares error model selector 520. As shown,input data 900 includes training data 902, which is then used to findthe coefficients for the output of modules 514-518 to obtainleast-squares errors between the input data and the predicted values.The resulting least-squares model is then applied to target range 904.

Although FIG. 5 illustrates particular components of intelligent anomalydetector 410, in different implementations, intelligent anomaly detector410 may include additional, fewer, or different components, or adifferent arrangement of components, than those illustrated. Forexample, ensemble learning anomaly detector 506 may include additionalstatistical modules for prediction (e.g., Kalman filtering module). Inanother example, pre-processor 504 may apply smoothing functions to rawdata before the data is further processed by ensemble learning anomalydetector 506.

FIG. 10 is a flow diagram of an exemplary process 1000 that isassociated with the intelligent anomaly detector 410. In someimplementations, process 1000 may be performed by NPO system 212.

As shown, process 1000 may include receiving user input and data (block1002). For example, intelligent anomaly detector 410 may receivespecific series of KPIs from data platform 404. The KPIs may include,for example, call drop rates, handover failure counts, jitters, etc. Inaddition to the data, intelligent anomaly detector 410 may receive userinput. For example, intelligent anomaly detector 410 may receive userselection of a particular module (among modules 514-520) to detectanomalies. In one implementation, intelligent anomaly detector 410 mayrun continually on a series of input data using the same selected moduleuntil the selection is changed.

Process 1000 may include removing extreme values from the input data(block 1004). For example, extreme value detector 510 may remove datapoints that are outliers. In one implementation, extreme value detector510 may identify data points that are outside of [Q₁−1.5 IQR, Q₃+1.5IQR] as being extreme and filter the data.

Process 1000 may further include determining if the filtered data iswhite noise (block 1006). For example, white-noise testing module 512may apply the Ljung-Box test to determine if the filtered data is whitenoise. If the filtered data is white noise (block 1006—YES), process1000 may perform the IQR detection test (block 1008). For example,process 1000 may apply IQR anomaly detector 508 (block 1008). Asdescribed above, IQR anomaly detector 508 may apply the interquartiletesting to the unfiltered data. to identify any points that are in theextreme ranges.

Returning to block 1006, if the filtered data is not white noise (block1006—NO), the data points are interrelated and may have detectablepatterns. Accordingly, process 1000 may apply the ensemble learninganomaly detection scheme (block 1011). In one embodiment, when applyingthe ensemble learning anomaly detection scheme, any of modules 514-520may be used. For instance, depending on user input, detecting anomaliesmay include using basic decomposition module 514 (block 1012), usingseasonal ARIMA module 516 (block 1014), using decomposition based ARIMAmodule 518 (block 1016), or using least-squares error model selector 520(block 1018), as already described above. Applying least-squares errorselector 520, however, requires first applying modules 514-518, and thenlearning an appropriate linear combination of the results of applyingmodules 514-518 (i.e., finding linear coefficients that minimize theleast-squares errors between the data and the predicted values). If thecoefficients are found, they can be stored, and the coefficients can beupdated as more data are received and applied.

After the output of either the IQR detection (block 1008) or theensemble learning anomaly detection (1011) is generated, the output maybe processed (block 1010). Depending on the user input, the output maybe graphed (e.g., along with the upper bound and lower bound) asfunction of time, alarms/alerts may be sent to operators, and/orappropriate actions may be taken as countermeasures (e.g., by componentsother than intelligent anomaly detector 410). For example, certainpackets may be re-routed, device parameters may be modified, antennatilt angles may be changed, or other actions may be taken ascountermeasures against detected anomalies or to prevent the anomaliesin the predicted values from emerging in real data.

When process 1000 is performed, intelligent anomaly detector 410 maypredict potential, future anomalies, even though no anomalies maycurrently exist. Because intelligent anomaly detector 410 is predictive,it can drive the network to take steps to prevent the anomalies beforethey occur.

As discussed above, NPO system 212 not only includes intelligent anomalydetector 410, but also includes network performance health (NPH) monitor412. NPH monitor 412 measures both user QoE and network performance.FIG. 11 illustrates an exemplary process 1100 that is associated withNPH monitor 412.

As shown, process 1100 may include receiving KPIs for a specified timeinterval (e.g., a day) (block 1102). For each node (e.g., eNodeB, cell,etc.), each received KPI is categorized (block 1104) based on itsrelevance to one of the following five categories: accessibility,retainability, availability, mobility and service integrity for aparticular type of network technology (e.g., LTE or VoLTE). Asub-process for handling the specific category of KPIs then classifies aparticular KPI as either lower-is-better or higher-is-better (block1106). “Lower-is-better” means that lower the value of a KPI, higher theperformance of the network component/element to which the KPI pertains.If the KPI is the higher-is-better type, then lower the value of theKPI, lower the performance of the network component/element to which theKPI pertains. Each sub-process is performed for each KPI, each of theKPI categories, and the specified time interval.

FIG. 12 illustrates exemplary logic of process 1100. As shown, lines 1-3indicate that the KPIs are processed for each network node (e.g., eNodeBor cell) and each category of KPI (i.e., availability, retainability,accessibility, mobility, service integrity). For each category andnetwork node to which the KPI pertains, each KPI is classified as eitherlower-is-better or higher-is-better. These operations are illustrated inFIG. 12, beginning with entries into for-loops over nodes, over each KPIcategory, and then over each KPI.

Returning to FIG. 11, at block 1108, if the KPI is lower-is-better(block 1108: YES), process 1100 may determine to which interquartile theKPI belongs (block 1110). If the KPI is lower than the lower bound(e.g., Q1−1.5 IQR) (block 1112: <LOWER BOUND), the base NPH score forthe node and the KPI is set to 100% (block 1114). If the KPI is greaterthan the upper bound (block 1112: >Q3+1.5 IQR), process 1100 may set theNPH score for the node and the KPI to 0% (block 1118). If the KPI isbetween the lower bound and the upper bound (block 1112: ininterquartile), process 1100 may calculate the base NPH score based onthe cumulative distribution function (CDF) for the KPI (block 1116).

In FIG. 12, lines 11-12 correspond to block 1114 of FIG. 11, lines 13-14correspond to block 1116, and lines 7-10 correspond to block 1116.Assuming that the base NPH score is denoted by Base.NPH(KPI, CATEGORY,NODE, TIME), determining the base NPH requires first determining acumulative distribution function for the particular KPI and determiningareas under/over the CDF. Given a series of KPI values, the cumulativedistribution function (CDF) may be first determined using well-knowntechniques. Next, an area under the curve (AUC) may be determined forthe CDF, in the range between an upper bound U (e.g., 97.5% threshold)and the particular KPI value. This can be expressed as:AUC _(KPI CATEGORY NODE)=∫_(KPI) ^(U) CDF(KPI)d(KPI)  (4)FIG. 13A illustrates the AUC, for a KPI classified as lower-is-better.In FIG. 13A, the AUC is the shaded area under 1302, ranging from U 1304to KPI 1306. The AUC can be normalized by the AUC in the range [L, U].The normalizing area, denoted as Perfect AUC, is given by:Perfect AUC _(KPI CATEGORY NODE)=∫_(L) ^(U) CDF(KPI)d(KPI)  (5)The base NPH score is computed in terms of the AUC and Perfect AUC as:Base.NPH(KPI,CATEGORY,NODE,TIME)=AUC _(KPI CATEGORY NODE)/Perfect AUC_(KPI CATEGORY NODE)  (6)

Returning to block 1108, if the KPI is not lower-is-better (block 1108:NO), process 1100 may determine to which interquartile the KPI belongs(block 1122). If the KPI is lower than the lower bound (e.g., Q1−1.5IQR) (block 1124: <LOWER BOUND), the base NPH score for the node and theKPI is set to 0% (block 1126). If the KPI is greater than the upperbound (block 1124: >Q3+1.5 IQR), process 1100 may set the NPH score forthe node and the KPI to 100% (block 1130). At block 1124, if the KPI isbetween the lower bound and the upper bound (block 1124: ininterquartile), process 1100 may calculate the base NPH score based onthe CDF for the KPIs (block 1128).

In FIG. 12, lines 21-22 correspond to block 1126 of FIG. 11 and lines23-24 correspond to block 1130, and lines 17-20 correspond to block1128. Assume that the base NPH is denoted by Base.NPH(KPI, CATEGORY,NODE, TIME). Computing the NPH score entails calculating an AUC and aPerfect AUC (a normalization factor) of the CDF for the KPI. Given aseries of KPI values, the CDF is first determined using well-knowntechniques. Then, the AUC is simply an area above the CDF, in the rangebetween the particular KPI and the lower bound U (e.g., 2.5% threshold).The AUC can be expressed as:AUC _(KPI CATEGORY NODE)=∫_(L) ^(KPI) CDF(KPI)d(KPI)  (7)FIG. 13B illustrates the AUC, for a KPI classified as higher-is-better.In FIG. 13B, the AUC is the shaded area above 1312, ranging from KPI1316 to L 1318. The perfect AUC is the area under the CDF, but rangingfrom the U to L. in accordance with expression (5). The base NPH scoreis still given by expression (6).

After performing actions in blocks 1114, 1116, 1118, 1126, 1128, or1130, process 1100 may arrive at block 1140, where it returns to block1106 to classify another KPI as lower-is-better or higher-is-better.Process 1100 may continue to perform actions in one or more of blocks1118-1130 and 1140, until all of the KPIs in the particular categoryhave been processed for calculating base NPH scores.

After the particular category of KPIs have been processed to generatethe base NPH scores (for each KPI, CATEGORY, NODE, and TIME), the baseNPH scores may be used to calculate correlation scores between the baseNPH scores and mean opinion scores (MOSs) that relate to user networkexperience. In this step, coefficients for the base NPH are selected tomaximize the correlation scores. The coefficients are updated throughback propagation learning for a convolutional neural network (CNN1)(block 1142 and lines 26-28 in FIG. 12). Furthermore,

$\begin{matrix}{{{Base} \cdot {{NPH}( {{NODE},{TIME}} )}} = {\sum\limits_{i = 1}^{\;}\lbrack {{Coeff}_{{Category}{(i)}} \cdot {Base} \cdot {{NPH}( {{CATEGORY},{NODE},{TIME}} )}} \rbrack}} & (9)\end{matrix}$In expression (8), the summation is performed over the KPIs, and theresulting base NPH score is no longer dependent on KPIs. The TIME inexpression (8) is 24 hours (daily). At block 1144, each coefficient (foreach CATEGORY) is updated by applying back propagation learning toanother convolutional neural network (CNN2) (see line 30 in FIG. 12). Inaddition,

$\begin{matrix}{{{Base} \cdot {{NPH}( {{CATEGPORY},{NODE},{TIME}} )}} = {\sum\limits_{j = 1}^{\;}\lbrack {{Coeff}_{{{Category}{(i)}}{{KPI}{(j)}}} \cdot {Base} \cdot {{NPH}( {{KPI}_{j},{CATEGORY},{NODE},{TIME}} )}} \rbrack}} & (8)\end{matrix}$In expression (9), the summation is performed over different categories,and the resulting base NPH score is no longer dependent on KPIcategories. The time period for each base NPH score is 24 hours (daily).

FIGS. 11 and 12 illustrate blocks/actions that relate to convolutionalneural networks. For example, at block 1142 and 1144 of FIG. 11 andlines 26-30, CNN1 and CNN2 are trained to obtain coefficients fordetermining base NPHs for each category and for the overall base NPH(which is derived by summing over the categories). The role of CNN1 andCNN2 in the context of determining Base.NPH(CATEGORY, NODE, TIME) andBase.NPH(NODE, TIME) is illustrated in FIG. 14. FIG. 14 depicts trainingCNN1 and CNN2 in the process associated with FIG. 11 and FIG. 12;

As shown in FIG. 14, different types of KPIs are used to generate NPHs.For example, FIG. 14 shows accessibility KPI 1411 and availability KPIs1415 as inputs. Other types of KPI are also input, but not illustrated(e.g., retainability KPIs, mobility KPI's, service integrity KPIs).These input KPIs span a specific time interval, as shown by tables 1421and 1425. For tables 1421 and 1425, each KPIs span M days. The KPIs arenext used to calculate base NPHs in accordance with expressions (6) and(7), denoted by block 1431 . . . and 1435. Thus, the output of blocks1431 . . . 1435 are daily base NPHs (for each KPI).

The output base NPH scores at blocks 1431 . . . 1435 are then correlatedwith MOS, to determine correlation coefficients. The correlationcoefficients are selected to maximize the correlation between the baseNPHs and MOS. Thereafter, the coefficients are updated in CNN1, via backpropagation training (see block 1452), and then used to output base NPHscores, each per category (e.g., availability, accessibility, etc.)(1454 and 1456).

The resulting base NPH scores are next used as input to CNN2. In amanner similar to that described above, correlation coefficients areselected to maximize the correlation score between MOS and the NPHs.Next, CNN2 is trained via back propagation, so as to update thecoefficients (block 1458). The output of CNN2 is the overall base NPHscores (over all the different categories).

FIG. 15 is a flow diagram of an exemplary process 1500 for determiningan NPH score for a grid element. Because there is no KPI that pertainsto a grid element, given a grid element, NPH scores must be calculatedbased on KPIs relating to network elements (e.g., cells). In oneimplementation, process 1500 may be performed by NPH monitor 412.

As shown, process 1500 may include, for a given grid element,identifying cells that overlap or intersect the grid element (block1502). For each cell that overlaps or intersects the grid element, ahandover count may be determined (block 1504).

Process 1500 may further include determining the base NPH for theselected grid element and the selected cell:Base.NPH(CELL,GRID,TIME)=Base.NPH(CELL,TIME)·Handover_(CELL GRID)/Handover_(GRID)  (10)In expression (10), NPH(CELL, TIME) denotes the base NPH score for aselected node (which is a cell in this implementation) and a gridelement intersecting with the cell. Base.NPH(CELL, TIME) denotes thedaily base NPH score in accordance with expression (9).Handover_(CELL GRID) denotes the number of handover for a selected celland a grid element, and Handover_(GRID) denotes the total number ofhandovers in the selected grid element. Essentially, the base NPH scorefor a selected cell and a selected grid element is found by normalizingthe base NPH score from expression (9) to the ratio of cell handovers inthe grid element to the total number of handovers in the grid element.Expression (10) may be evaluated for each cell intersecting with theselected grid element.

Process 1500 may also include summing the Base.NPH(CELL, GRID) over thecells intersecting with the grid element (block 1508). Summing the baseNPH scores over the cells removes the dependent variable CELL, leavingthe base NPH score that depends on the grid element.

FIG. 16A illustrates an exemplary process 1600 for determining bi-weeklybase NPH scores for network nodes. In some implementations, process 1600may be performed by NPH monitor 412. FIGS. 17A and 17B illustrateprocessing daily base NPH scores to obtain bi-weekly NPH scores inaccordance with process 1600. Although FIGS. 16A, 17A and 17B are basedon obtaining NPH scores for the time interval of two weeks, in otherimplementations, the scores may be obtained for different time intervals(e.g., monthly NPH scores, yearly NPH scores, two-day NPH scores, weeklyNPH scores, etc.).

As shown, process 1600 may include averaging the base NPH score computedfrom expression (6). That is, after base NPH scores are calculated foreach KPI, each category of KPIs, and each node daily (i.e.,Base.NPH(KPI, CATEGORY, NODE, DAILY)). The scores may be averaged overtwo weeks. As noted above, in other implementations, a different timeinterval (e.g., a month) may be selected. Accordingly, for process 1600,Base.NPH(KPI,CATEGORY,NODE,BI-WEEKLY)=Two week average ofBase.NPH(KPI,CATEGORY,NODE,DAILY)  (11)FIGS. 17A and 17B illustrate the averaging performed in expression (11),for eNodeBs and cells. In FIG. 17A, blocks 1702 and 1704 illustrate theaveraging for eNodeBs. In FIG. 17B, blocks 1712 and 1714 illustrate theaveraging for cells.

Process 1600 may further include summing the Base.NPH(KPI, CATEGORY,NODE, BI-WEEKLY) (obtained from expression (11) over different KPIs(block 1604), which yields Base.NPH(CATEGORY, NODE, BI-WEEKLY). Toperform the sum, it is necessary to account for the coefficientspreviously provided in expression (8). That is,Base.NPH(CATEGORY,NODE,BI-WEEKLY)=SUM(Base.NPH(KPI,CATEGORY,NODE,BI-WEEKLY)·Coeff_(CATORY KPI))  (12)

The dependent variable KPI is removed from the base NPHs throughexpression (12). The base NPH scores resulting from expression (12) canthen be summed over different categories, to finally obtainBase.NPH(NODE, BI-WEEKLY) (block 1606). Accordingly,Base.NPH(NODE)=SUM(Base.NPH(CATEGORY,NODE,BI-WEEKLY)·Coeff_(CATEGORY))  (13)

FIG. 16B illustrates an exemplary process 1620 for determining bi-weeklybase NPH scores for grid elements. In some implementations, process 1620may be performed by NPH monitor 412. FIG. 17C illustrates processingdaily base NPH scores to obtain bi-weekly NPH scores for grid elementsin accordance with process 1620.

As shown, process 1620 may include, for a given grid element,determining cells that intersect with the grid element (block 1622), anddetermining the total number of handovers for the grid element (block1624). The total number of handovers may be obtained from collected dataon handovers for different cells.

Process 1620 may further include, for each cell in the grid element,determining bi-weekly base NPH scores based on the total number ofhandovers and the number of handovers per cell (block 1626). Morespecifically, for each cell intersecting with the grid element, process1620 evaluates:Base.NPH(CELL,GRID)=Base.NPH(CELL,BI-WEEKLY)·Handover(CELL,GRID)/Handover(GRID)  (14)Expression (14) is also illustrated in FIG. 17C. As shown, block 1722,which represents the base NPH score for each grid element, receives baseNPH scores for cells, from block 1712.

After evaluating expression (14), process 1620 may determine thebi-weekly base NPH score for each grid element using the base NPH(CELL,GRID) obtained from expression (14) (block 1628). FIG. 17C illustratesthe computation at block 1724. The result of expression (14) is thensummed over the cells that intersect the particular grid element, toremove the dependence of the base NPH score on cells.

FIG. 18 is a flow diagram of a process 1800 for determining final dailyand bi-weekly NPH scores for nodes and grid elements using performanceadjusters. FIG. 19 illustrates the processing of FIG. 18. In otherimplementations, the final NPH scores may be determined for timeintervals other than two-weeks. Process 1800 may include, for each node(e.g., eNodeB, cell, etc.), applying an activation function andnormalization to obtain a performance adjuster (block 1802).

For example, assume that an activation function is a sigmoid function,which is defined asSigmoid(x)=(1+exp(x))⁻¹  (15)

To obtain a performance adjuster for a grid element, first defineTARGET_(MAX)=1, TARGET_(MIN)=½, B1=2, B2=5, and B3=3. Then assign theactivity level (activation) of each grid element as a sigmoid functionof these variables:

$\begin{matrix}{{{Sigmoid}({GRID})} = {{Sigmoid}( {{{SUM}( {{B\; 1},{B\; 2\mspace{14mu}\ldots}}\mspace{11mu} )} \cdot {( {{B\;{1 \cdot {SCORE}_{{RSRP}\mspace{14mu}{GRID}}}} + {B\;{2 \cdot {SCORE}_{{SINR}\mspace{14mu}{GRID}}}} + {B\;{3 \cdot {SCORE}_{{HO}\mspace{14mu}{GRID}}}}} )/2}} )}} & (16)\end{matrix}$

In expression (16), SCORE_(RSRP GRID), SCORE_(SINR GRID) andSCORE_(HO GRID) denote: a reference signal received power score; asignal-to-interference-plus-noise ratio score; and a handover score.These scores are obtained from different components in data collectionsystem 402. FIG. 19 illustrates evaluating expression (16) as blocks1902 (determining cell density weight) and block 1904 (applying thecalculated cell density weight to a sigmoid function).

According to expression (16), the sigmoid function can attain differentvalues. Let Sigmoid_(MAX) and Sigmoid_(MIN) be the maximum and theminimum of these values. Then, an NPH score can be rescaled from therange [Sigmoid_(MIN), Sigmoid_(MAX)] to the range [TARGET_(MIN),TARGET_(MAX)], by multiplying by the following factor:

$\begin{matrix}{{{PA}({GRID})} = {\lbrack {{{Sigmoid}({GRID})} - {SIgmoid}_{MIN}} \rbrack/{\quad{{\lbrack {{Sigmoid}_{{MA}\; X} - {Sigmoid}_{MIN}} \rbrack \cdot \lbrack {{TRAGET}_{{MA}\; X} - {TARGET}_{MIN}} \rbrack} + {TARGET}_{MIN}}}}} & (17)\end{matrix}$

In expression (17), the rescaling factor is the PA (performanceadjuster), which is also illustrated by block 1906 (normalization) andblocks 1912 (performance adjusters for grid elements) in FIG. 19. OncePAs are obtained based on expression (17), the PAs may be used tocompute final NPH scores (block 1804). The computation may be performedin accordance with the following expression:FINAL NPH(GRID,TIME)=Base.NPH(GRID,TIME)·PA(GRID)  (18)

For different network elements (e.g., cell or eNodeB), performanceadjusters can be obtained in a manner similar to that described abovefor each grid element. FIG. 19 illustrates computing performanceadjusters for cells and eNodeBs at blocks 1910 and 1912. For each node,FINAL NPH(NODE,TIME)=Base.NPH(NODE,TIME)·PA(NODE)  (19)In one implementation, expressions (18) and (19) are evaluated for TIMEset to DAILY or BI-WEEKLY. In other implementations, different timeperiods may be used.

Various NPH scores (e.g., those in expressions (8)-(14), (18), and (19))relate to healthiness of different aspects of a given network.Furthermore, some of the above NPH scores relate to user QoE (e.g.,expression (8) and (9)). As explained above, a given set of NPH scoresmay pertain to a particular technology. For example, in oneimplementation, NPH scores are derived for LTE or Voice-over-LTE(VoLTE).

FIG. 20 illustrates exemplary final NPH scores and final bi-weekly NPHscores for different types of nodes as tables. In one implementation,each score in tables 2002-2012 may be determined by NPH monitor 412 andstored in data visualization and automation 408 (e.g., networkreconfiguration system 414). Tables 2002-2006 illustrate the dailyscores for grid elements, eNodeBs, and cells, respectively, and tables1908-1912 illustrate bi-weekly scores for grid elements, eNodeBs, andcells, respectively. Each table may include six types of NPH scores forLTE and VoLTE: overall NPH, accessibility NPH, retainability NPH,service integrity NPH, availability NPH, and mobility NPH. To obtainthese NPH scores, for each technology (i.e., LTE and VoLTE), NPH monitor412 may apply the results of process 1100 (daily base NPH scores foreNodeB and cells), process 1500 (daily base NPH scores for gridelements), processes 1600 and 1620 (bi-weekly base NPH scores), andprocess 1800 (obtaining and applying performance adjusters).

FIG. 21 illustrates an exemplary decision tree that is associated withNPH scores. In one implementation, decision tree 2100 may be provided tousers of NPO system 212 via data visualization and automation system408. As shown in FIG. 21, the first component of decision tree 2100 isthe network technology. In tree 2100, NET technology 2102 indicates thatthe particular technology that tree 2100 addresses is VoLTE. For VoLTE,daily NPH 2104 is shown as performance score of 53.21.

Tree 2100 branches into various categories associated with thetechnology, such as accessibility, mobility, retainability, serviceintegrity, and availability. In FIG. 21, the degree to which each branchcontributes to the overall score (i.e., weight) is indicated by itsthickness. For example, in FIG. 21, the branches to the VoLTEaccessibility and the VoLTE retainability are thicker than thoseconnected to other categories, indicating their relative contribution tothe overall NPH score at NET technology level 2102.

Each of the branches to the five subcategories can be selected via auser interface. In FIG. 21, the VoLTE retainability is selected, andhence, its NPH score 2108 is shown (i.e., NPH(CATEGORY, DAILY)), alongwith the weight of 0.4 associated with the branch. The selection ofVoLTE retainability also leads to display of its sub-branches toindividual KPIs that contribute to the overall VoLTE retainability NPHscore. In FIG. 21, five such values are illustrated (e.g., SIP calldrop, SIP abnormality, VOLTE LTE, etc.).

In tree 2100, various branches are shaded differently to indicate theextent to which the NPH score can be considered “good” or “bad.” Forexample, a darker color may indicate greater negative impact of eachscore to the overall NPH score. For example, the branch from VoLTEretainability to SIP_ABNORMALITY is shaded darker than other branches,to indicate the negative impact of the KPI to the VoLTE retainabilityNPH score. In other implementations, different color scheme may be usedto indicate the impact of each scores of the branches (e.g., a red orgreen color gradient indicating a negative or positive impact).

FIG. 22 illustrates an exemplary network reconfiguration system 414 in anetwork. As shown, network reconfiguration system 414 may include areconfiguration manager 2202 and one or more network node(s) 116. Insome implementations, reconfiguration system 414 may include relativelyfew reconfiguration managers and many network nodes. In otherimplementations, system 414 may include distributed configurationmanagers 2202 over multiple devices/platforms. In still otherimplementations, all the functionalities of reconfiguration manager 2202may be included in components installed in network nodes 116.Reconfiguration manager 2202 may receive data from intelligent anomalydetector 410 and NPH monitor 412 and issue commands/data to network node116. Network node 116 receives commands/data from reconfigurationmanager 2202 and controls network components in accordance with thecommands/data.

As shown, reconfiguration manager 2202 may include a reconfigurationmanager interface 2212, a database 2214, and a server 2216.Reconfiguration manager interface 2212 acts as a gateway between manager2202 and other network entities, such as intelligent anomaly detector410, NPH monitor 412, administrator devices, user devices, etc. Forexample, reconfiguration manager interface 2212 may receive informationregarding anomalies from intelligent anomaly detector 410 and NPH scoresfrom NPH monitor 412. In addition, interface 2212 may receive controlcommands from operator/user devices (e.g., via web interface), accessinformation, and provide information to user devices. When interface2212 receives information from external network entities, interface 2212may store the information in database 2214.

Database 2214 may store data on detected and predicted anomalies, NPHscores, control settings, and/or other information received throughinterface 2212. Database 2214 may also retrieve and/or modify such datain response to user commands. In some implementations, database 2214 mayalso store rules for modifying network device parameters based on thepredicted/detected anomalies and NPH scores.

Server 2216 may determine appropriate actions for network nodes 116based on predicted or detected anomalies and NPH scores, and dispatchcommands to network nodes 116. In some implementations, the actions maybe determined based on programmed procedures. In other implementations,server 2216 may retrieve relevant rules from database 2214 and apply therules to data on anomalies and/or NPH scores. Server 2216 may record theresults of issuing the commands in database 2214, as well as thecommands themselves.

For example, when server 2216 determines that there is likely to be anetwork anomaly in specific cells, server 2216 may obtain rulesregarding anomalies detected or predicted for cell traffic. If the ruleindicates that the traffic problem can be eliminated by load-balancingthe traffic between two cells, server 2216 may issue commands tospecific network nodes 116 (e.g., to direct handoffs).

As also shown, network node 116 may include a client 2222, a localdatabase 2224, and a controller 2226. Client 2222 may receive commandsand/or data from server 2216, and store the received information inlocal database 2224. Database 2224 may store local copies of receiveddata/commands through client 2222. In some implementations, database2224 may also store readings (e.g., device parameters) collectedlocally. For example, if network node 116 is implemented as a router,database 2224 may collect traffic data and/or related statistics.

Controller 2226 may modify engineering parameters for network componentsand/or devices at node 116 in response to the received data and/orcommands at client 2222. For example, according to one implementation inwhich network node 116 is implemented as a base station, controller 2226may receive a command to modify direction of the antenna. In otherimplementations (e.g., in network components other than base station,such as a router, serving gateway, etc.), controller 2226 may adjustother types of network performance/operating parameters, such as loadbalancing parameter, memory utilization parameter, bandwidth allocationbased on traffic type, antenna related parameters, routing paths, etc.

In this specification, various preferred embodiments have been describedwith reference to the accompanying drawings. It will be evident thatmodifications and changes may be made thereto, and additionalembodiments may be implemented, without departing from the broader scopeof the invention as set forth in the claims that follow. Thespecification and drawings are accordingly to be regarded in anillustrative rather than restrictive sense.

In the above, while a series of blocks have been described with regardto the processes illustrated in FIGS. 10, 11, 15, 16A, 16B, and 18, theorder of the blocks may be modified in other implementations. Inaddition, non-dependent blocks may represent blocks that can beperformed in parallel.

It will be apparent that aspects described herein may be implemented inmany different forms of software, firmware, and hardware in theimplementations illustrated in the figures. The actual software code orspecialized control hardware used to implement aspects does not limitthe invention. Thus, the operation and behavior of the aspects weredescribed without reference to the specific software code—it beingunderstood that software and control hardware can be designed toimplement the aspects based on the description herein.

Further, certain portions of the implementations have been described as“logic” that performs one or more functions. This logic may includehardware, such as a processor, a microprocessor, an application specificintegrated circuit, or a field programmable gate array, software, or acombination of hardware and software.

To the extent the aforementioned embodiments collect, store or employpersonal information provided by individuals, it should be understoodthat such information shall be used in accordance with all applicablelaws concerning protection of personal information. The collection,storage and use of such information may be subject to consent of theindividual to such activity, for example, through well known “opt-in” or“opt-out” processes as may be appropriate for the situation and type ofinformation. Storage and use of personal information may be in anappropriately secure manner reflective of the type of information, forexample, through various encryption and anonymization techniques forparticularly sensitive information.

No element, block, or instruction used in the present application shouldbe construed as critical or essential to the implementations describedherein unless explicitly described as such. Also, as used herein, thearticles “a,” “an,” and “the” are intended to include one or more items.Further, the phrase “based on” is intended to mean “based, at least inpart, on” unless explicitly stated otherwise.

What is claimed is:
 1. A device comprising: a communication interface;memory to store instructions; one or more processors to: collect, from awireless network and through the communication interface, first datapertaining to nodes in the wireless network; perform, for each datum ofthe first data, learning by: calculating a first base NetworkPerformance Health (NPH) score; and applying first deep learning to afirst neural network among a plurality of neural networks to updatefirst coefficients for correlating the first base NPH score to a meanopinion scores, wherein each datum of the first data belongs to one of aplurality of categories, and wherein the categories include at least oneof service availability, service integrity, service accessibility,mobility, or retainability; obtain second base NPH scores by using thefirst coefficients, wherein the second base NPH scores indicate userQuality of Experience (QoE) with the wireless network; obtain a value ofan operating parameter of one of the nodes based the first NPH scoresand the second NPH scores; and send, through the communicationinterface, the value of the operating parameter to the one of the nodes,wherein the one of the nodes is configured to set the operatingparameter to the value to optimize operation of the wireless network. 2.The device of claim 1, wherein the first data includes a series of keyperformance indicator values.
 3. The device of claim 1, wherein the onenode includes a base station.
 4. The device of claim 1, wherein when theone or more processors calculate the first base NPH score, the one ormore processors are to: determine if the datum is inside a range whereat least about 50% of the first data is found; and set a first base NPHscore, among the first base NPH scores, to a predetermined value if thedatum is not in the range.
 5. The device of claim 1, wherein theplurality of neural networks include convolutional neural networks. 6.The device of claim 1, wherein each datum of the first data isassociated with one of a plurality of time intervals, and wherein eachof the time intervals is about one hour to 24 hours.
 7. The device ofclaim 1, wherein when the one or more processors perform learning, theone or more processors are to: determine a cumulative distributionfunction (CDF) for the first data; obtain an area under a curve for theCDF over a first range; and normalize the area under the curve bydividing the area under the curve by an area under the curve of the CDFcomputed over a second range.
 8. The device of claim 1, wherein when theone or more processors calculate the first NPH score, the one or moreprocessors are to: determine whether increasing a value of the datumimproves performance of the wireless network or decreasing the value ofthe datum improves performance of the wireless network.
 9. The device ofclaim 1, wherein the first data includes a series of key performanceindicator values, and wherein the one node includes a base station. 10.The device of claim 1, wherein the one or more processors are furtherto: determine a third base NPH score for a geographical area based onbase NPH scores that are functions of the nodes, the data, and time. 11.The device of claim 10, wherein when the one or more processorsdetermine the third base NPH score, the one or more processors are to:determine a subset of nodes, among the nodes, that intersect ageographical area, and sum normalized values of third base NPH scoresover the subset of nodes to obtain the third base NPH score.
 12. Amethod comprising: collecting, from a wireless network, first datapertaining to nodes in the wireless network; performing learning, foreach datum of the first data, by: calculating a first base NetworkPerformance Health (NPH) score; and applying first deep learning to afirst neural network among a plurality of neural networks to updatefirst coefficients for correlating the first base NPH score to a meanopinion scores, wherein each datum of the first data belongs to one of aplurality of categories, and wherein the categories include at least oneof service availability, service integrity, service accessibility,mobility, or retainability; obtaining second base NPH scores by usingthe first coefficients, wherein the second base NPH scores indicate userQuality of Experience (QoE) with the wireless network; obtaining a valueof an operating parameter of one of the nodes based the first NPH scoresand the second NPH scores; and sending the value of the operatingparameter to the one of the nodes, wherein the node is configured to setthe operating parameter to the value to optimize operation of thewireless network.
 13. The method of claim 12, wherein the first dataincludes a series of key performance indicator values.
 14. The method ofclaim 12, wherein the one node includes a base station.
 15. The methodof claim 12, wherein calculating the first base NPH scores includes:determining if the datum is inside a range where at least about 50% ofthe first data is found; and setting a first base NPH score, among thefirst base NPH scores, to a predetermined value if the datum is not inthe range.
 16. The method claim 12, wherein the plurality of neuralnetworks include convolutional neural networks.
 17. The method of claim12, wherein each datum of the first data is associated with one of timeintervals, and wherein each of the time intervals is about one hour to24 hours.
 18. The method of claim 12, wherein the performing includes:determining a cumulative distribution function (CDF) for the first data;obtain an area under a curve for the CDF over a first range; andnormalize the area under the curve by dividing the area under the curveby an area under the curve of the CDF computed over a second range. 19.The method of claim 12, wherein calculating the first NPH scoreincludes: determining whether increasing a value of the datum improvesperformance of the wireless network or decreasing the value of the datumimproves performance of the wireless network.
 20. The method of claim12, wherein the first data includes a series of key performanceindicator values, and wherein the one node includes a base station.